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Abstract 

Student engagement in classroom learning was conceptualised as a balance 
between two attributes. In order for students to be engaged, the tasks 
expected of them should be commensurate with their ability to complete 
these tasks. That is, a balance between their learning capabilities and the 
expectations of their learning. Of interest was the proportion of students with 
this balance. 

First, 194 Years Eight to Twelve students were interviewed by two 
researchers on six aspects of the expectations of their learning and five 
aspects of learning capabilities. Second, 1760 Years Eight to Twelve students 
responded to 15 self-report items about the expectations of their learning and 
12 self-report rating scale items about their learning capabilities. Rasch 
model common-person test equating methods were then applied separately to 
the data from the two sources. The proportions of students with ‘equivalent’ 
learning capabilities and expectations of learning scores were approximately 
80% for the two samples. 

The results support the theoretical basis for the Capabilities Expectations 
Model of engagement. Significantly, methods for estimating the proportion 
of students engaged in their classroom learning were presented and assessed. 
Instrumentation reliability within and between methods was evidenced. 


This research was funded by a large-scale Linkage grant from the Australian Research Council. 
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Background 

Models of latent traits often postulate relations between two or more variables or sub-constructs. The 
relation between variables can be represented mathematically as an algebraic expression or graphically 
as a plot on a Cartesian plane (two variables). The following sections examine the theory underpinning 
two of these models - The Experience Fluctuation Model of flow and the Expectations Capabilities 
Model of student engagement in classroom learning. These are followed by a brief overview of test 
equating to show how the Rasch model can be used to estimate equivalence between data from two 
measures to garner evidence for the construct validity of a latent trait model. 

The Experience Fluctuation Model of flow 

According to Csikszentmihalyi (1990), when people described optimal experiences (situations which 
are highly enjoyable), they often use the term flow. Flow is to the . . spontaneous, seemingly 
effortless aspect of such experiences” (Csikszentmihalyi & Schneider , 2000, p. 97). A recurring 
aspect of descriptions of flow is the balance between perceived high levels of challenge and high 
levels of skill - the task is demanding but the enjoyment of the experience also derives from having the 
skills necessary to complete the task (Massimini, Csikszentmihalyi & Carli, 1988). From this 
perspective, flow is a function of challenge and skills. Challenge and skill have been measured and 
above average levels on these two dimensions are assumed to indicate flow. One approach to 
measuring flow uses the Experience Sampling Method (ESM) (Hekter, Schmidt & Csikszentmihalyi, 
2007) in which data is collected from an individual at regular intervals over a prolonged period. For 
example every 30 minutes over several days. The mean level of perceived experiences is calculated for 
this person and then his/her experiences are classified relative to this level. Figure 1 below, the 
Experience Fluctuation Model, provides a visual representation of flow by plotting challenge and skill 
on a Cartesian plane. Other categories or conditions of experiences such as anxiety, apathy and 
relaxation can also be represented by the Experience Fluctuation Model (Csikszentmihalyi & FeFevre, 
1989; Nakamura, 1988). 



Figure 1 The Experience Fluctuation Model 
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The Expectations Capabilities Model of student engagement in learning 

Cavanagh, Kennish and Sturgess (2008), found reference to flow theory in the literature on student 
engagement and they noted similarities between flow and engagement. They proposed that a 
student who is engaged within a particular situation is expected to have a balance between the 
perceived level of the challenge being faced and his/her perceived capability to meet the incumbent 
requirements” (Cavanagh, Kennish & Sturgess, 2008, p. 15). The flow theory terms of ‘challenge’ and 
‘skills’ were replaced by’ expectations of learning’ and ‘learning capabilities’, terms that more 
accurately describe attributes of students and learning. The Expectations Capabilities Model of 
student engagement in classroom learning is visually represented in Figure 2 below. 


Zone of 
Engagement in 
Learning 


> 

LEARNING CAPABILITIES 

Figure 2 The Expectations Capabilities Model 

The diagonal zone is a region where the capability of a student to complete a task is commensurate 
with the difficulty of what is expected of the student. Students in this zone at a particular time and in a 
particular situation, have a balance between what they are capable of doing and what is required of 
them. When high expectations are placed on highly capable students, it is anticipated they will learn 
more and more quickly. Less capable students can still be engaged provided the expectations of them 
are not too high. It is anticipated these students will learn less and at a slower rate. 

While the Model has application for explaining phenomena such as engagement in learning in an 
illustrative way, it is speculative in the absence of operational definitions and measures. Overcoming 
this theoretical deficiency requires measures of expectations of learning and of learning capabilities, 
and a technique for identifying students who have a balance between these two aspects of their 
learning. Previous research into the model has developed operational definitions, constructed measures 
and collected observational and self-report data (Cavanagh, 2011; Cavanagh & Kennish, 2009). 
However, quantifying the balance between expectations and capabilities has received less attention. 
There is a need to establish criteria for demonstration of equivalence between expectations and 
capabilities scores. And then, to ascertain in particular situations, the proportion of students with 
scores meeting these criteria. 

Test equating using the Rasch model 

There are often occasions when comparisons are required of estimates of ability or attitudes made at 
different times, with different populations or using different instruments. Traditionally, Classical Test 
Theory is applied with student performance presented as the proportion of items answered correctly. 
This requires all the students to be tested with the same items. Similarly estimating item difficulty 
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requires all the items to be administered to the same students. These are serious limitations when 
comparing test scores for the same students at different times, at the same time for students in different 
year cohorts, from different tests, or from a combination of these conditions. Alternatively, the Rasch 
model approach creates measures which are invariant over time and between groups of persons. Item 
difficulty is not dependent on the persons tested and the scores of the persons are not dependent on the 
subset of items administered. These properties of data that fit the Rasch model enable test scores to be 
compared provided there are some common items, or some common persons sitting both tests. Further 
advantages of the approach are the plotting of person ability estimates and item difficulty estimates on 
linear scales, and estimation of a standard error for each person’s score and each item’s difficulty. 
Interval data are available for subsequent mathematical operations and when statistics such as t- values 
are calculated, the level of significance is calculated from the person score errors. 

The current study used Rasch model test equating to examine equivalence between paired-sample 
expectations scores and capabilities scores. The purpose was to determine the degree of equivalence 
on the assumption that the scores were related but not necessarily measuring the same sub-construct. 

Research questions 

Two methods of common-person test equating were applied One method used t-tests to estimate the 
number of students with equivalent scores (95% confidence interval) (RUMM2030 Andrich, Sheridan 
& Luo, 2012). The other used a graphical process in which control lines for a 95% confidence band 
were constructed from the standard errors in individual scores and then the number of coordinates 
within the band was counted (see Bond & Fox, 2007). Both methods were applied to researcher- 
completed data and also to student self-report data. The research questions were: 

1. For the researcher-completed data, what proportion of students have t- values within a 95% 
confidence interval? 

2. For the self-report data, what proportion of students have t- values within a 95% confidence 
interval? and 

3. What proportion of students have expectations and capabilities score coordinates within a 95% 
confidence band around a diagonal through the mean estimates? 


Methodology 

The researcher-completed data were obtained from interviews of 194 secondary school students. The 
sample comprised Years Eight to Twelve students, boys and girls, city and country students. They 
were asked about their experiences in either English, Mathematics, Science or Society and 
Environment classes. The researchers rated each student on six-point scales for six aspects of 
expectations of learning and five aspects of learning capabilities (see Cavangh, 2009). The student 
self-report data were obtained from 1760 students responding to 27 items on a three-point scale - 
strongly agree, agree and disagree, (see Cavanagh, 2012). Fifteen of the items were about expectations 
of learning and twelve were about learning capabilities. 

Data from both sources were entered into RUMM2030 (Andrich, Sheridan & Luo, 2012) and separate 
analyses were performed. Two sub-tests were created in each analysis, one for expectations of learning 
and one for learning capabilities. Person scores (estimated in logits) and standard errors were 
estimated for each person for each sub-test. 

The first equating procedure involved estimation of t-values for each student’s two sub-test scores, 
plotting the distribution of these values and then identifying the students with values exceeding the 5% 
significance level. This procedure was applied to the researcher-completed data and then the self- 
report data. The second equating procedure commenced by calculating the mean scores for the sub- 
tests and adjusting these so there was a common mean score. The coordinates of the adjusted sub-test 
scores were then plotted on a Cartesian plane (expectations of learning scores on the vertical axis and 
learning capabilities scores on the horizontal axis), using Excel. The standard errors in each pair of 
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student sub-test scores were then used to construct the control lines for a band of 95% confidence. A 
visual inspection of the scatterplot was then made to count the number of students with scores within 
the band. 

Results 

For the researcher-completed data (N= 194), the mean Rasch calibrated scores were 0.52 logits 
(standard deviation 1.61 logits) for expectations of learning and 1.40 logits (standard deviation 1.74 
logits) for learning capabilities. The person separation index (proportion of variance in the calibrated 
scores considered true) was 0.84 for expectations of learning and 0.89 for learning capabilities. 

For self-report data (N= 1760), the mean Rasch calibrated scores were 0.53 logits (standard deviation 
1.61 logits) for expectations of learning, and 0.66 logits (standard deviation 1.61 logits) for learning 
capabilities. The person separation index (proportion of variance in the calibrated scores considered 
true) was 0.87 for expectations of learning and 0.84 for learning capabilities. 

The researcher-completed data fitted the Rasch Rating Scale model very well - low residual values 
(<+ 2.5) and high Chi Square probability values (> 0.05). The student self-report data fitted less well 
due to dependency between several of the items. 


T-value procedure 

Figure 3 below shows the distribution of t-values for the researcher-completed data (N= 194). 17.4% 
(34 students) lie outside of the 95% confidence interval. 


Distribution of t-test values Caps; Expect 



t-te st va I □ e 


Figure 3 Distribution of t-values (N= 194) 

Figure 4 below shows the distribution of t-values for the self-report data (N= 1760). 17.9% (312 
students) lie outside of the 95% confidence interval. 
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Distribution of t-test values Caps; Expect 



t-test value 


Figure 4 Distribution of t-values (N= 1760) 


Confidence band procedure 

Figures 5 and 6 are scatterplots of the expectations and capabilities scores for the researcher- 
completed data and the self-report data. 



Figure 5 Scatterplot of expectations and capabilities scores (N= 194) 
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Approximately 15 (18%) of the 194 coordinates are outside of the 95% confidence bands. The 
approximation is due to the difficulty in counting coordinates close to the control lines. 


Common Item Linking Plot 



Expectations Estimates 


Figure 6 Scatterplot of expectations and capabilities scores (N= 1760) 


The density of coordinates makes counting numbers of scores outside of the confidence band difficult 
and inaccurate. 


Conclusion 

The proportions of students with equivalent (95% confidence interval) in the researcher-completed 
sample and the self-report sample were very similar - respectively 82.6% and 82.1%. The confidence 
bands procedure produced a similar value for the researcher-completed sample. These findings are 
evidence for the representation of student engagement in learning presented in the Expectations 
Capabilities Model. 

Approximately 80% of the students measured would lie within the diagonal zone of engagement. This 
value provides a reference point for further research using measurement and the Expectations 
Capabilities Model. 

Both Rasch model test equating methods calibrate scores on an interval scale and then paired scores 
including standard errors for each score, are used for the analysis. This can be compared with 
traditional methods that use raw scores and correlational analyses. The requirements for data fitting 
the Rasch model are stringent and the results of subsequent analyses can be interpreted with 
confidence. 
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